GPUI frame benchmarks: interleaved A/B per platform, scene-equivalence gate, Cap editor A/B by richiemcilroy · Pull Request #3886 · CapSoftware/Cap

richiemcilroy · 2026-11-06T13:14:56Z

Benchmarks for Cap's GPUI patch stack, run per platform with base and head interleaved on the same machine, plus a gate that fails if a patch changes what gets drawn.

flowchart LR
  P[PR touches apps/desktop-gpui/patches] --> W1[gpui-bench.yml<br/>macOS · Windows · Linux]
  P --> W2[gpui-app-bench.yml<br/>macOS]
  W1 --> B[build base + head<br/>from one checkout path]
  B --> M[interleaved Criterion rounds<br/>+ Window::draw per frame]
  B --> D[scene digests<br/>frame by frame]
  B --> T[gpui / platform crate tests]
  W2 --> A[real cap-gpui, base + head<br/>editor timeline bench on the<br/>reference recording]

Frame benchmarks (`apps/desktop-gpui/gpui-benches`)

14 workloads modelled on Cap windows, each drawing real frames through Window::draw with the platform text system; on macOS each frame is also submitted to Metal through the headless renderer:
- cards_{64,256,1024}: bordered, shadowed cards with two labels each.
- text_rows_static_{50,200} and text_rows_changing_{50,200}: label rows, the second with a timestamp that changes every frame.
- paragraphs_{12,48}: wrapped text in columns.
- recordings_scroll: a 10 000-row uniform_list scrolling.
- hover_cards: pointer moves across 256 hoverable cards.
- cached_panels: 40 .cached() views, one of which changes per frame.
- primitives_{1024,8192}: raw quads, shadows and paths.
The crate lives in Cap. scripts/gpui-bench-build.sh copies it into the prepared zed checkout, so it builds against the exact patched gpui and Zed's lockfile.
Each benchmark registers through a small runner (run in benches/frame.rs) rather than #[gpui::bench], so the headless renderer can be wrapped.

Measuring (`scripts/gpui-bench-compare.py`)

Base and head are built from the same checkout path, swapped in turn. With identical sources, builds from two different paths differ by a few percent (paths end up in symbol hashes).
Runs alternate base, head, head, base, … and pair each round's Criterion median; a change is called faster or slower only when every round agrees.
A second table reports Window::draw alone, from GPUI's own frame report. On macOS, light frames are bound by the headless GPU handing back instance buffers, so wall time hides CPU work.

Scene equivalence (`scripts/gpui-scene-digest.py`)

With GPUI_SCENE_DIGEST=<file>, the runner hashes every scene a benchmark submits before passing it to the real renderer. The hash covers all primitives in the order the renderer receives them, plus the pixels uploaded into every atlas tile a sprite shows.
CI compares base and head at every (benchmark, frame) position both builds reached, and fails on any difference. It also fails when a benchmark is missing, draws fewer than 16 frames on either side, or (on macOS) digests nothing.
Changing a single bit of every rasterized glyph makes 704 of 800 positions differ, and the gate fails.
Only macOS has a headless renderer, so the step reports a skip elsewhere.

Real app (`.github/workflows/gpui-app-bench.yml`, `scripts/gpui-app-bench.py`)

Builds cap-gpui against base and head, then runs CAP_GPUI_AUTO_TIMELINE_BENCH=1 (src/editor_bench.rs) on CapSoftware/cap-performance-fixtures' reference recording. Every run gets a fresh profile and a fresh copy of the project.
Compares each scenario's median draw over 5 interleaved rounds. The editor decodes video while it plays, so means and tails swing with the decoder.

Profiling (`.github/workflows/gpui-profile.yml`)

On pushes to gpui-profile/**, runs each benchmark under callgrind on Linux and uploads the annotated self and inclusive costs.

Who can trigger it

The benchmark and test jobs run on pushes, workflow_dispatch, and pull requests from this repository. Fork pull requests don't start them.

Tests

gpui-bench.yml also runs cargo test -p gpui --lib --tests and the platform crate's tests (gpui_macos, gpui_windows, or gpui_wgpu on Vulkan) against the head stack on all three platforms.

🤖 Generated with Claude Code

… and Linux Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ng the toolchain Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…fault 1.95.0) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The frame benchmarks register through a small runner instead of #[gpui::bench] so that, with GPUI_SCENE_DIGEST set, every scene a benchmark submits is hashed before reaching the real headless renderer. scripts/gpui-scene-digest.py compares those digests between base and head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sandchest-com · 2026-11-06T13:15:30Z

Sandchest reviewed e9fad53: nothing exploitable found in this change.

_{Sandchest reviews every change for vulnerabilities an attacker could use · 2 credits · Review details}

superagent-security

Superagent found 1 security concern(s).

superagent-security · 2026-11-06T13:16:47Z

+    paths:
+      - "apps/desktop-gpui/patches/**"
+      - "scripts/gpui-app-bench.py"
+      - ".github/workflows/gpui-app-bench.yml"


P2: Untrusted fork PRs can repeatedly queue multi-hour GPUI benchmark jobs by editing benchmark-only paths

New PR workflows trigger 120-150 min macOS/Windows/Linux benchmark jobs when only the workflow, its scripts, or gpui-benches crate change.

Gate self-file/benchmark-triggered jobs to internal PRs or a maintainer-approved label/environment; keep fork runs bounded.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior. <file name=".github/workflows/gpui-app-bench.yml"> <violation number="1" location=".github/workflows/gpui-app-bench.yml:15"> <priority>P2</priority> <title>Untrusted fork PRs can repeatedly queue multi-hour GPUI benchmark jobs by editing benchmark-only paths</title> <evidence>The new `pull_request` trigger lists the workflow itself and `scripts/gpui-app-bench.py` as paths (lines 11-15), so a fork PR that changes nothing but this workflow or its runner script starts the `editor` job on `blacksmith-6vcpu-macos-15` with `timeout-minutes: 150` (lines 36-39). That job runs `node scripts/setup.js`, prepares head and base GPUI checkouts, builds `cap-gpui` twice (lines 107-112), and then runs the real editor timeline benchmark for `--rounds 5` interleaved A/B rounds over the reference recording (lines 126-139). The new `.github/workflows/gpui-bench.yml` has the same shape: paths at lines 12-17 include `apps/desktop-gpui/gpui-benches/**`, `scripts/gpui-bench-*`, and the workflow file itself, and each matching PR push queues three `bench` jobs (150-minute timeout) plus three `test` jobs (120-minute timeout) across macOS, Windows, and Linux. Neither workflow carries a `github.event.pull_request.head.repo.full_name == github.repository` condition, a required label, or an `environment:` approval gate; `concurrency` with `cancel-in-progress: true` only cancels overlapping runs and does not stop sequential pushes from each consuming a full run. The 150-minute and 120-minute timeouts are the worst-case per-run cost, and each run performs two full Cargo builds plus the interleaved measurement workload, so a fork contributor can repeatedly trigger expensive shared-runner work with a trivial edit to benchmark-only files.</evidence> <recommendation>Restrict the self-file/benchmark-only triggers to internal pull requests, e.g. add `if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository` to the `bench`/`test`/`editor` jobs (or drop the workflow/script/crate paths from `pull_request.paths` and run them only from `workflow_dispatch` / `push` on trusted branches). For fork PRs, keep the benchmark suite reachable only through a maintainer-applied label or an `environment:` approval, and keep a short bounded validation run for workflow-only changes.</recommendation> </violation> </file>

Fixed in e3a8c7f: the bench and test jobs in gpui-bench.yml and the editor job in gpui-app-bench.yml now run only when github.event.pull_request.head.repo.full_name == github.repository (or on push/workflow_dispatch). Fork PRs no longer start them; maintainers can still measure a fork's patch with workflow_dispatch.

Correctly fixed: gating the long-running bench, test, and editor jobs to internal PRs via github.event.pull_request.head.repo.full_name == github.repository (and push/workflow_dispatch) prevents untrusted forks from automatically queuing expensive benchmark runs, while preserving maintainer flexibility. Dismissing.

capy-ai

Capy found 2 potential issues (2 medium).

View 1 other finding in Capy.

capy-ai · 2026-11-06T13:18:02Z

+    if not base or not head:
+        print("No scenes were digested (this platform has no headless renderer); skipping.")
+        return
+    compared = sorted(set(base) & set(head))
+    differing = [key for key in compared if base[key] != head[key]]


Scene equivalence succeeds when scene coverage is missing

Medium Severity

On macOS, the patched current_headless_renderer() returns None when Metal initialization fails, so a benchmark can exit successfully without producing any digests. If only head encounters that failure, this branch skips the gate despite base having rendered scenes. Partial loss is also ignored because only intersecting frame keys are compared: head can stop submitting frames after an unchanged prefix and still report identical scenes. Restrict skipping to explicitly unsupported platforms and require expected benchmark/frame coverage before comparing hashes.

Fixed in e3a8c7f. An empty digest on macOS now fails instead of skipping (skipping only happens when neither side digests anything on a non-macOS runner). Every benchmark must draw at least 16 frames on both sides, and every (benchmark, frame) position both builds reached must be present and equal, so a head that stops submitting a workload fails. Positions beyond what both reached aren't compared, because a faster build legitimately draws more frames in Criterion's fixed time.

greptile-apps · 2026-11-06T13:19:09Z

+    if not base or not head:
+        print("No scenes were digested (this platform has no headless renderer); skipping.")
+        return
+    compared = sorted(set(base) & set(head))
+    differing = [key for key in compared if base[key] != head[key]]


Missing frames pass the gate

If the head stops submitting a workload or submits fewer frames, the comparison drops those positions because it checks only shared keys. If either side submits nothing, it skips entirely. A patch that stops drawing can therefore pass the scene check. Require matching benchmark/frame keys, and allow empty results only on platforms explicitly known to lack a renderer.

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/gpui-scene-digest.py Line: 70-74 Comment: **Missing frames pass the gate** If the head stops submitting a workload or submits fewer frames, the comparison drops those positions because it checks only shared keys. If either side submits nothing, it skips entirely. A patch that stops drawing can therefore pass the scene check. Require matching benchmark/frame keys, and allow empty results only on platforms explicitly known to lack a renderer. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Same fix as above (e3a8c7f): a missing benchmark or one with fewer than 16 frames on either side fails, and every frame position both builds reached must be present and identical.

greptile-apps · 2026-11-06T13:19:09Z

+cd "$zed_dir"
+messages="$out_dir/cargo-messages.json"
+cargo "+${GPUI_BENCH_TOOLCHAIN:-1.95.0}" bench -p gpui_cap_benches --bench frame --no-run --message-format json > "$messages"


Relative output paths break

The helper creates out_dir from the caller's directory, then changes to zed_dir before opening cargo-messages.json and copying the executable. Calling scripts/gpui-bench-build.sh ../zed-cap ./bench-out therefore writes to a different location or fails before Cargo runs. Resolve out_dir to an absolute path before changing directories.

Prompt To Fix With AI

This is a comment left during a code review. Path: scripts/gpui-bench-build.sh Line: 36-38 Comment: **Relative output paths break** The helper creates `out_dir` from the caller's directory, then changes to `zed_dir` before opening `cargo-messages.json` and copying the executable. Calling `scripts/gpui-bench-build.sh ../zed-cap ./bench-out` therefore writes to a different location or fails before Cargo runs. Resolve `out_dir` to an absolute path before changing directories. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fixed in e9fad53: out_dir is resolved to an absolute path right after it's created, before the script changes directory.

greptile-apps · 2026-11-06T13:19:09Z

+/// cover the warm-up frames every run draws, whatever Criterion's timing.
+const DIGEST_FRAMES: usize = 64;
+
+/// Wraps the platform's headless renderer and hashes each scene it submits.


Comments repeat the code

This comment restates what SceneDigestRenderer does. The comments above cards and primitives do the same, and the pattern appears elsewhere. AGENTS.md requires comments to explain non-obvious context rather than narrate the code. Remove these descriptions while keeping comments that explain choices such as the measured build-path effects. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: apps/desktop-gpui/gpui-benches/benches/frame.rs Line: 585 Comment: **Comments repeat the code** This comment restates what `SceneDigestRenderer` does. The comments above `cards` and `primitives` do the same, and the pattern appears elsewhere. `AGENTS.md` requires comments to explain non-obvious context rather than narrate the code. Remove these descriptions while keeping comments that explain choices such as the measured build-path effects. This repository requirement must be satisfied before merging. **Context Used:** AGENTS.md ([source](https://github.com/capsoftware/cap/blob/main/AGENTS.md)) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fixed in e9fad53 (and in zed-perf.patch on #3887): removed the comments that describe the code, keeping only ones that record non-obvious context (e.g. why base and head build from one path, why tile pixels must be hashed, why a tile id can be reused).

…the benchmark runners - The digest now includes a hash of the pixels uploaded into every atlas tile a sprite shows, so a rasterization change fails the gate even when tile sizes and ids stay the same. - Each benchmark must draw at least 16 frames on both sides and every frame both builds reached must match; an empty run on macOS fails instead of skipping. Builds may still draw different numbers of frames in Criterion's fixed time. - The benchmark and test jobs only run for pull requests from this repository; forks can still be measured with workflow_dispatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel · 2026-11-06T13:57:10Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
cap-status	Ignored	Preview	Nov 6, 2026 2:07pm UTC

capy-ai

Capy found no issues.

View 1 other finding in Capy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

capy-ai

Capy found no issues.

View 1 other finding in Capy.

richiemcilroy and others added 11 commits November 6, 2026 10:08

Add GPUI frame benchmarks with interleaved A/B runs on macOS, Windows…

b128989

… and Linux Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Bound Criterion samples per benchmark round

944b5eb

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add an interleaved A/B run of the Cap editor's draw benchmark

7c8eb20

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add a callgrind profile of the GPUI frame benchmarks

de368f5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Build both benchmark sides from one checkout path

138ebad

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Compare the editor benchmark by median draw over five rounds

02c5a76

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Resolve benchmark executables before changing directory; allow choosi…

e0b4c01

…ng the toolchain Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Report each benchmark size on its own, and compare per-frame draw time

e519b4d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Profile 15 s of frames per benchmark

5e28ae0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Let local runs pick the benchmark toolchain (GPUI_BENCH_TOOLCHAIN, de…

1825c6c

…fault 1.95.0) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

superagent-security Bot reviewed Nov 6, 2026

View reviewed changes

capy-ai Bot reviewed Nov 6, 2026

View reviewed changes

greptile-apps Bot reviewed Nov 6, 2026

View reviewed changes

richiemcilroy mentioned this pull request Nov 6, 2026

GPUI: faster frames (balanced BoundsTree, cheaper text painting and scene finish) #3887

Merged

vercel Bot deployed to Preview – cap-web November 6, 2026 13:59

View deployment

capy-ai Bot reviewed Nov 6, 2026

View reviewed changes

Fix relative bench output paths; trim comments to non-obvious context

e9fad53

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

capy-ai Bot reviewed Nov 6, 2026

View reviewed changes

richiemcilroy merged commit e446eb3 into main Nov 6, 2026
68 checks passed

himanshu748 mentioned this pull request Nov 8, 2026

perf(gpui): resolve complex shape overlays in one pass with A/B benchmarks #3945

Open

Conversation

richiemcilroy commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Frame benchmarks (apps/desktop-gpui/gpui-benches)

Measuring (scripts/gpui-bench-compare.py)

Scene equivalence (scripts/gpui-scene-digest.py)

Real app (.github/workflows/gpui-app-bench.yml, scripts/gpui-app-bench.py)

Profiling (.github/workflows/gpui-profile.yml)

Who can trigger it

Tests

Uh oh!

sandchest-com Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

superagent-security Bot left a comment

Choose a reason for hiding this comment

Uh oh!

superagent-security Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

superagent-security Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

capy-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

capy-ai Bot Nov 6, 2026

Choose a reason for hiding this comment

Scene equivalence succeeds when scene coverage is missing

Uh oh!

richiemcilroy Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

richiemcilroy Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

vercel Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

capy-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

capy-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

richiemcilroy commented Nov 6, 2026 •

edited

Loading

Frame benchmarks (`apps/desktop-gpui/gpui-benches`)

Measuring (`scripts/gpui-bench-compare.py`)

Scene equivalence (`scripts/gpui-scene-digest.py`)

Real app (`.github/workflows/gpui-app-bench.yml`, `scripts/gpui-app-bench.py`)

Profiling (`.github/workflows/gpui-profile.yml`)

sandchest-com Bot commented Nov 6, 2026 •

edited

Loading

vercel Bot commented Nov 6, 2026 •

edited

Loading